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1.0 Introduction 

The rapid pace of technological change and the move toward “open systems” is making the pro- 
cess of acquiring systems much more complex. Traditionally, functional and performance require- 
ments have been carefully described for systems to be acquired and the systems usually have 
come from a single vendor. The process worked as long as the requirements remained nearly 
static and systems changed slowly over their life time. There generally has been no need for a 
requirement to provide measurements and performance monitoring to see that requirements were 
met over the long term. Measurements that were available were often left over from development. 

In the future the requirements for many systems are expected to change more quickly, and parts of 
the systems, acquired from multiple vendors, will evolve to meet those changing needs. There is a 
desire to ask for life-time measurements of systems in request for proposals (RFPs) when systems 
are being acquired. Thus, there is a need for measurements and performance monitoring as an 
integral part of the system to ensure that requirements are met over the long term after acceptance. 

This paper gives a strawman proposal for a framework for presenting a common set of metrics for 
supercomputers, workstations, file servers, mass storage systems, and the networks that intercon- 
nect them. Production control and database systems are also included. Though other applications 
and third party software systems are not addressed, it is important to measure them as well. 

The capability to integrate measurements from all these components from different vendors, and 
from the third party software systems has been recognized and there are efforts to standardize a 
framework to do this. The measurement activity falls into the domain of management standards. 
Standards work is ongoing for Open Systems Interconnection (OSI) systems management; AT&T, 
Digital and Hewlett-Packard are developing management systems based on this architecture even 
though it is not finished. Other efforts include the Storage System Management Sub-committee of 
the Mass Storage System Working Group and the UNIX International Performance Management 
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Working Group [1]. In addition, there are the Open Systems Foundation’s Distributed Manage- 
ment Environment and the Object Management Group. A paper comparing the OSI systems man- 
agement model and the Object Management Group model has been written [2]. Though most of 
the standards effort has been on the mechanisms for gathering and reporting measurements, we 
expect to cooperate with these standards making efforts. The work reported here is ongoing. 

The IBM world has had a capability for measurement for various IBM systems since the 1970’s 
and different vendors have been able to develop tools for analyzing and viewing these measure- 
ments. Since IBM was the only vendor, the user groups were able to lobby IBM for the kinds of 
measurements needed. However, in the UNIX world of multiple vendors, a common set of mea- 
surements will not be as easy to get. 

In this paper we distinguish between metric and measurement. A measurement is a quantity that 
is directly measured while a metric is a quantity that can be derived from a set of measurements. 
Our focus is on using low level vendor specific measurements to support a set of higher level met- 
rics that are common across a variety of vendors. The set of measurements to support the common 
metrics should in general be the minimum that is provided. Most systems should also make avail- 
able measurements of unique aspects of the system that are not covered by the common set. For 
example, measurements on vectorization and hit ratios for memory hierarchies may not be in the 
common set of metrics but such measurements are desired. 

2.0 Uses for Measurements 

Measurements of systems are, of course, useful in many other ways than just to support system acquisition. 
They can be used to support day-to-day operations, management decisions and planning, and performance 
monitoring. The following are seven types of uses we have identified: 

(1) distributed computing system scheduling, 

(2) fire-fighting - solve immediate problems to provide acceptable response time and resource 
allocation to all processes, 

(3) tuning systems for current workloads, 

(4) capacity planning, 

(5) allocating resources, 

(6) looking for trends and characterizing workloads, 

(7) verifying system strategies are working or assumptions about workloads are valid, e.g. lo- 
cality of reference, 

(8) validating accounting reports. 

In analyzing how measurements are used, the following three points are very important. (1) For 
fire-fighting and tuning, a systems administrator must be able to link a particular “event” to a set 
of user commands.The systems administrator should be able to know when a resource is respond- 
ing slowly and which process is causing the problem. We stress that it is important to be able to 
link particular events of interest back to user commands though we know that it is sometimes dif- 
ficult. (2) Process as well as system-wide measurements are needed. (3) Accurate time stamps or 
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other timing information is necessary so that various independent measurements can be correlated 
with each other as a system is observed over time. 


3.0 Measurement Collection Techniques 

It is also understood that taking measurements and collecting them cause overhead and may in 
extreme cases affect the performance of the systems measured; this is not specifically addressed in 
this paper. However, data can be collected at various levels of detail depending on how much 
overhead is involved. The most complete level of measurement is a log or trace of each transac- 
tion or event. The next level of measurement is a set of counters that produce a histogram, which 
is an approximation to the distribution, of the metric of interest. The least detailed level of mea- 
surement is a simple counter from which the average, variance, maximum and minimum of the 
metric of interest can be derived. The level of measurement for any component depends on the 
overhead associated with the workload. When possible, the ability to selectively choose a differ- 
ent measurement level allows users of a system to manage how much overhead is given to mea- 
surement activities. Another way of managing the overhead associated with measuring a system is 
to sample a measurement at some interval that is frequent enough to observe interesting behavior 
but with reduced overhead. The sampling rate should be adjustable. 

For measurements to be useful, they must be well documented It must be clear exactly what is 
being measured. The documentation should specify how much overhead is involved, what tech- 
nique is being used to generate the measurement, and if there are user selectable parameters such 
as a sample rate or an enable/disable switch. Information about how a system is configured must 
either be statically defined or recorded along with a set of measurements. 
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Figure 1: Model of Network Computing System 


4.0 Model of Distributed High Performance Computing Systems 

In Figure 1 we present a model of a distributed high performance computing system. The model 
identifies the five highest level functions of external input sources to indicate the collection of 
data for processing in the system, distribution for the network among components, supercom- 
puter processing for high performance computing, storage for distributed mass storage, and 
visualization for user support processing. The distributed characteristics of this model are not 
depicted specifically but one can think of NASA’s EOS system as the basis for this model. The 
other high performance computing systems listed at the bottom of the figure will all have similar 
models. 

The five model functions are made up of various hardware and software system components. The 
hardware system components include supercomputers, mainframes, workstations, mass storage 
devices, file servers, networks, input machines and other network devices such as disk arrays. The 
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software system components include operating systems (OS) (includes file system and protocols), 
mass storage systems (MSS), database systems (DBMS), production control systems, third party 
software and user applications. Below the system component level are lower level building blocks 
to measure. These are the hardware building blocks such as CPU, memory, memory interconnec- 
tion, disk, tape, terminal I/O, recorder/drive, robotics box, channel/controller, network interface, 
router and external I/O. The software building blocks are dependent on the particular system soft- 
ware component. For an operating system there are process management (scheduler/queues, con- 
text switches), I/O system (buffers, cache, queues), memory management (allocation, swapping, 
queues, paging, caches), file system, protocols, interprocess communications and other operating 
system services. For a mass storage system there is each module in the Mass Storage Reference 
Model (MSRM). For an application there are user defined modules, operating system compo- 
nents and various hardware building blocks used by the application. For a database there are 
indexes, tables, stored procedures, logs, locks, transactions and users. The software building 
blocks are not yet completely identified. 


Figure 2 illustrates this three level hierarchy of metrics. The abstract metrics at the base of the 
pyramid are a list of generic metrics that are used at all three levels. The eye represents the need to 
have comprehensive and uniform observations at all levels. 




Figure 2. Hierarchical Levels 
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In Figure 3, the levels are expanded to show how higher level objects are composed of lower level 
objects. Three model functions are connected to system components and some of the system com- 
ponents are connected to lists of building blocks. Space does not permit a full expansion of all 
functions and components in this figure. 

The common metrics for objects at the lowest level are derived from vendor measurements. And 
the metrics for higher level objects are derived from combinations of metrics for lower level 
objects. Thus we expect that the derivation of the lowest level metrics will be vendor specific but 
that higher level metrics will be vendor independent. We have not dealt with the issue of how to 
attach derivations to the metrics. We have not evaluated the difficulty for vendors to implement 
measurements to support the framework, nor have we tried to gauge the relative importance of the 
various metrics. 
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Figure. 3 Expanded Hierarchy 


The use of common metrics is intended to provide vendors and other system developers a frame- 
work that can be used to design measurements as an integral part of the system that they deliver. 


544 






Too often measurements are used only to verify that a system is operating correctly and are insuf- 
ficient for understanding the performance of the system especially when it is interconnected as a 
component of a larger system. 


5.0 Abstract Metrics 

The following list of abstract metrics are used to observe any object in the hierarchy by specify- 
ing an instance of the metric for the object: 

1 Utilization, Capacity, Idle 

2 Throughput 

3 Response Time 1 , Delay, Expansion Factor 2 

4 Waiting Time 

5 Service Time 

a. Bitfile Size, Packet Size, Computation Requirement 

b. Speed of Device 

6 Queue Length 

7 Number of Jobs, Bitfiles, Packets 

8 Routing, Branching Probabilities for Jobs Paths, Reuse, Age 

9 Hit Ratios, Effectiveness of Strategies 
(data migration, locality of reference) 

10 Error Rates 

All of these metrics are commonly used except for 8 and 9. Branching probabilities are useful for 
modeling systems. 

At the bottom of the hierarchy the specific metric for each object is given in terms of characteris- 
tics of the object, such as mips and mflops for CPU throughput metrics. In addition at higher lev- 
els, users will want to specify metrics in terms of the workload of the system, e.g. satellite images 
processed per second through ail model level functions for the NASA EOS. 


6.0 Metric Tables 


The following pages contain tables of metrics for the objects within the hierarchy. The left hand 


column in each table has the list of abstract metrics. The other columns have instances of the cor- 


responding metric for the object at the top of the column. The tables are generally sparse since this 


1. Response Time - Service Time + Wait Time + Other Time 

2. Expansion Factor wall clock time in shared system / wall clock time in dedicated system, 
which often can be approximated by wall clock time in shared system / CPU time 
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work is still ongoing and we invite help in completing the tables, adding more objects to the hier- 
archy and adding more abstract metrics. 


abstract 

metric 


utilization, 

capacity, 

idle 


throughput 


Processing 
and Storage 


Processing 
and Input 


Input and 
Storage 


Input, Processing, 
Storage 



Mops per bit 
stored 

Mops per bitfile 
stored 


Mops per bit 
input 


Input bits per bit 
stored 


response time, 
delay, 

expansion factor 


waiting time 


service time: 
job size, 
device speed 


queue length 


number of jobs 


routing paths, 
reuse, age, 
branching 
probabilities 


hit ratios, effec- 
tiveness of 
strategies 


error rates 



Bitfiles processed 
through all functions 
per second 


Response time through 
all functional compo- 
nents 


Bitfile routes through 
all functional compo- 
nents 



Table 1: Model Level - Across Functions 


Table 1 has metrics for the overall system where the objects being observed are combinations of 
model level functions. At this level, the metric instances are suggestions since they will depend on 
what the system does and will be defined by the users of the system. 

Tables 2, 3 and 4 have the metrics for storage and some of its lower level components and build- 
ing blocks. Tables 5 through 9 have the metrics for supercomputer processing and some of its 
lower level objects. Table 10 has the metrics for distribution (networks) and some of its lower 
level objects 

In conclusion, we have presented a strawman proposal for a framework for presenting a common 
set of metrics across many systems and we have listed some of the metrics. This work is ongoing 
and we invite participation from users, vendors and system developers. 
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abstract metric 


storage 


mass storage device 


mass storage 
reference model 


utilization, 

capacity, 

idle 

% space used 
#B a ,#0,#M total by 
class b or storage device 

% space used 
% fragmentation 

# bitfiles by class 
bitfiles/media 
bits/bitfile by class 

throughput 

{B 0 M)/sec access 0 
by class or storage 
device 

bits/sec accessed 
media/sec accessed 

bitfiles/sec accessed 

response time, 
delay, 

expansion factor 

{B O M) response time 
by class or storage 
device or overall 

{B M) response time 
by class 

Bitfile response time 
by class 

waiting time 

*d 


* 

service time: 
job size, 
device speed 

* 


* 

queue length 



length at various 
model modules 

number of jobs 




routing paths, reuse, 
age, branching prob- 
abilities 

# accesses vs. storage 
device 

{B O M) vs. age vs. 
# accesses 

#media vs.#accesses 
#media vs. # age 
#media vs. age vs. 

# accesses 

# bitfile vs. # accesses 

# bitfile vs. # age 

# bitfile vs. age vs. 

# accesses 

hit ratios, effectiveness 
of strategies 



migration policy met- 
ric 

hit ratios 

error rates 

BER overall, by device 
failure by device 

BER, failures 



T^ble 2: Stor ag e - Svstem ComoonentS 

O ' n f JL 


a B- Bits; O - Bitfiles; M = Media 

b. class - {media type, bitfile size, access type, user, user process, user defined) 

c. accesses - reads, writes, deletes 

d. Asterisk implies that the metric is the obvious one in this context 


System Components not included in this configuration: 

1 . workstation or mainframe for controlling mass storage device 

2. database for meta-data about the stored bitfiles 
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abstract 

metric 

utilization, 

capacity, 

idle 


throughput 


response time 8 , 
delay, 

expansion factor 


waiting time 

service time: 
job size, 
device speed 

queue length 

number of jobs 

routing paths, 
reuse, age, 
branching 
probabilities 


tape 

% space used/tape 
% free tapes 


# tapes vs. 

# accesses 15 

# tapes vs. age 

# tapes vs. age vs. 

# accesses 


recorder, 
tape drive 

disk arm/ 
platters 

robot 

% time reading 
% time writing 
% time scanning 
% idle 

% time reading 
% time writing 
% time seeking 
% free space or 
fragmented 
(int/ext) for 
platters 

% time in use 

bits read/sec 
bits write/sec 
mounts/sec 

bits read/sec 
bits write/sec 
seeks/sec 

# requests/sec 

includes (posi- 
tioning) start, 
stop, scan, 
read/write 
delays 

includes read / 
write, seek, 
rotation delays 

includes start, stop 
(positioning) 
delays 



hit ratios, effec- arm movement distance/request 

tiveness of distance/seek 

strategies 

error rates BER each tape failures/time int failures/int. robot failures/int 


BER for all tapes 


BER for platters 


Table 3: Storage - Building Blocks - Hardware 

a. response time - service time + waiting time + other factors associated with using resource 

b. accesses - reads, writes, deletes 
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abstract 

metric 


physical volume 
repository 


bit file mover 


storage server 


bit file server 


utilization, 

capacity, 

idle 

% time in use 




throughput 

bitfiles/sec 

accessed 

bitfiles/sec 

accessed 

requests/sec 

requests/sec 

response time, 
delay, 

expansion factor 

*a 

* 

* 

* 

waiting time 

* 

* 

* 

* 

service time: 
job size, 
device speed 

* 

* 

* 

* 

queue length 

♦ 

* 

* 

* 

number of jobs 


♦ 

* 

♦ 

routing paths, 
reuse, age, 
branching 
probabilities 



# {B 0} vs. age 
vs. size vs. 
accesses 


hit ratios, effec- 
tiveness of 
strategies 



migration/cach- 
ing policy 


error rates 






Table 4: Storage - Building Blocks - Software 


a Asterisk implies that the metric is the obvious one in this context. 
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abstract 

metric 


utilization, 

capacity, 

idle 


throughput 


response time, 
delay, 

expansion factor 


waiting time 


service time: 
job size, 
device speed 


queue length 


number of jobs 


routing paths, 
reuse, age, 
branching 
probabilities 


hit ratios, effec- 
tiveness of 
strategies 


error rates 


Supercomputer 

Processing 


% to users 
% to system 
% to idle 


Supercomputer 


% to users 
% to system 
% to idle 


mops, mips 
mflops 


Operating 

System 


Application 
CPU, mem, 10 


% to system % to application 

% holding on locks % to system 
vectorization 
speedup 



processes/ sec 
system calls/sec 
interrupts/sec 
- all by class 


response time for 
all processes 
expansion factor 
for all processes 


waiting time for all 
processes 


CPU burst time vs. 
memory size 


mflops 

particles/sec 


response time for 
application 



CPU time 
memory size 
logical reads, 
writes 



process path proba- 
bilities for I/O 
devices 




page hit ratio 
swaps 
system calls 


Table 5: Supercomputer Processing - System Components 
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abstract 

metric 


utilization, 

capacity, 

idle 


throughput 


response time, 
delay, expan- 
sion factor 


waiting time 


service time: 
job size, 
device speed 


queue length 


number of jobs 


routing paths, 
reuse, age, 
branching 
probabilities 


hit ratios, effec- 
tiveness of 
strategies 


error rates 


CPU 


% time issuing inst 
% time holding issue 
% time vect or para 
% vector {ops, inst} 
vector length 


ops/inst 
mops, mips 
mflops 


Memory 


% time issuing 
read or write 
(% free space or 
fragmented) 


{Bytes, Words} 
read/s, write/s 
by type 


SSD* 


% time issuing 
read or write 
(% free space or 
fragmented) 


reads/sec 

writes/sec 


disk arm/ 
platters 


% time reading 
% time writing 
% time seeking 
% free space or 
fragmented (int/ 
ext) for platters 


bits read/sec 
bits write/sec 
seeks/sec 



o time waiting on 
functional units 


hardware specified 


waiting time/ref 
hold issue/ref 
contention/ref 


hardware speci- 
fied 



hardware speci- 
fied 


includes read / 
write, seek, 
rotation delays 



instruction mix: 
% {ops, inst} by 
instr class 


instr cache hit ratio 
memory cache hit 
ratio 


page hit ratio 


device cache hit 



arm movement 
distance/seek 


failures/interval 
BER for platters 


Table 6: Supercomputer - Building Blocks - Hardware 


a. SSD = solid state device 
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abstract metric 


Channel/Controller 


Terminal I/O 


utilization, % time busy 

capacity, 

idle 

throughput bits/sec by device* characters/sec 

channel ops/sec 

response time, 
delay, 

expansion factor 

waiting time 

service time: 
job size, 
device speed 

queue length 

number of jobs 

routing paths, reuse, age, bits vs. device 

branching probabilities 

hit ratios, effectiveness of 
strategies 

error rates 

Table 7: Supercomputer - Building Blocks - Hardware 

a. device - {SSD, disk) 
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abstract metric 


CPU Management 


Memory 

Management 


I/O System 


utilization, 

capacity, 

idle 

% time to user 
% time to idle 
% time to system 

% space used 
% space fragmented 

% buffer space used 

throughput 

context switches/sec by 
class 

processes/sec 

allocations per sec 
swaps per second by 
memory size 
pages per sec 

logical & physical read / 
write per sec by bits, 
device 

response time, 
delay, 

expansion factor 




waiting time 

WT 

WT 

WT 

service time: 
job size, 
device speed 

CPU burst time per pro- 
cess 

memory size by process 
memory residency time 

time for service by logi- 
cal, physical 
I/O 

queue length 

QL of CPU queue(s) 

QL of Memory 
queue(s) 

QL of device queues 

number of jobs 


# jobs in memory 


routing paths, 
reuse, age, 
branching proba- 
bilities 




hit ratios, effective- 
ness of strategies 



I/O buffer hit ratio by 
read, write 

error rates 





Table 8: Operating System - Building Blocks - Software 
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abstract metric 


File System 


Interprocess 

Communication 


Other OS Services 


utilization, 

capacity, 

idle 

throughput 

response time, 
delay, 

expansion factor 

waiting time 

service time: 
job size, 
device speed 

queue length 

number of jobs 

routing paths, reuse, 
age, branching 
probabilities 

hit ratios, effective- 
ness of strategies 

error rates 



Table 9: Operating System - Building Blocks - Software 
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abstract 

metric 


utilization, 

capacity, 

idle 


throughput 


response time, 
delay, 

expansion factor 


waiting time 


service time: 
job size, 
device speed 


queue length 


number of jobs 


routing paths, 
reuse, age, 
branching 
probabilities 


hit ratios, effec- 
tiveness of 
strategies 


error rates 


Distribution 



bits/s 

bits/s vs. path 
objects/s 
objects/s vs. path 
by class 4 



Networks 

Operating 

System: 

Protocols 

% time used 
bits/packet 

bits/object 

packets/object 

bits/s 
packets/s 
by class 

bits/s 
pkt/s 
objects/s 
by class 

by class and object 
size 

by class and 
object size 

by class and object 
size 

by class and 
object size 

by class and object 
size 

by class and 
object size 

send/receive 

queues 

send/receive 

queues 


relative use of paths 



BER, failures 


retrans/sec 


timeouts 

failures 


Routers/ 

Network 

Interfaces 










Table 10: Distribution - System Components 

a. class - {protocol used, path, user, process, send/receive} 
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